1. Field of the Invention
The present invention pertains to typing recognition systems and methods, and more particularly to recognition of typing in air or on a relatively smooth surface that provides less tactile feedback than conventional mechanical keyboards.
2. The Related Art
Typists generally employ various combinations of two typing techniques: hunt and peck and touch typing. When hunting and pecking, the typist visually searches for the key center and strikes the key with the index or middle finger. When touch typing, the fingers initially rest on home row keys, each finger is responsible for striking a certain column of keys and the typist is discouraged from looking down at the keys. The contours and depression of mechanical keys provide strong tactile feedback that helps typists keep their fingers aligned with the key layout. The finger motions of touch typists are ballistic rather than guided by a slow visual search, making touch typing faster than hunt and peck. However, even skilled touch typists occasionally fall back on hunt and peck to find rarely-used punctuation or command keys at the periphery of the key layout.
Many touchscreen devices display pop-up or soft keyboards meant to be activated by lightly tapping a displayed button or key symbol with a finger or stylus. Touch typing is considered impractical on such devices for several reasons: a shrunken key layout may have a key spacing too small for each finger to be aligned with its own key column, the smooth screen surface provides no tactile feedback of finger/key alignment as keys are struck, and most touchscreens cannot accurately report finger positions when touched by more than one finger at a time. Such temporal touch overlap often occurs when typing a quick burst of keys with both hands, holding the finger on modifier keys while striking normal keys, or attempting to rest the hands. Thus users of touchscreen key layouts have had to fall back on a slow, visual search for one key at a time.
Since touchscreen and touch keyboard users are expected to visually aim for the center of each key, typing recognition software for touch surfaces can use one of two simple, nearly equivalent methods to decide which key is being touched. Like the present invention, these methods apply to devices that report touch coordinates interpolated over a fine grid of sensors rather than devices that place a single large sensor under the center of each key. In the first method, described in U.S. patent application Ser. No. 09/236,513 by Westerman and Elias, the system computes for each key the distance from key center to the sensed touch location. The software then selects the key nearest the finger touch. In the second method, described in U.S. Pat. No. 5,463,388 to Boie et al., the software establishes a rectangle or bounding box around each key and decides which, if any, bounding box the reported touch coordinates lie within. The former method requires less computation, and the latter method allows simpler control over individual key shape and guard bands between keys, but both methods essentially report the key nearest to the finger touch, independent of past touches. Hence we refer to them as xe2x80x98nearest keyxe2x80x99 recognizers.
Unlike touchscreens, the multi-touch surface (MTS) described by Westerman and Elias in Ser. No. 09/236,513 can handle resting hands and temporal finger overlap during quick typing bursts. Since the MTS sensing technology is fully scalable, an MTS can easily be built large enough for a full-size QWERTY key layout. The only remaining barrier to fast touch typing on an MTS is the lack of tactile feedback. While it is possible to add either textures or compressibility to an MTS to enhance tactile feedback, there are two good reasons to keep the surface firm and smooth. First, any textures added to the surface to indicate key centers can potentially interfere with smooth sliding across the surface during multi-finger pointing and dragging operations. Second, the MTS proximity sensors actually allow zero-force typing by sensing the presence of a fingertip on the surface whether or not the finger applies noticeable downward pressure to the surface. Zero-force typing reduces the strain on finger muscles and tendons as each key is touched.
Without rich tactile feedback, the hands and individual fingers of an MTS touch typist tend to drift out of perfect alignment with the keys. Typists can limit the hand drift by anchoring their palms in home position on the surface, but many keystrokes will still be slightly off center due to drift and reach errors by individual fingers. Such hand drift and erroneous finger placements wreak havoc with the simple xe2x80x98nearest keyxe2x80x99 recognizers disclosed in the related touchscreen and touch keyboard art. For example, if the hand alignment with respect to the key layout drifts by half a key-spacing (xcx9c9 mm or xe2x85x9cxe2x80x3), all keystrokes may land halfway between adjacent keys. A xe2x80x98nearest keyxe2x80x99 recognizer is left to choose one of the two adjacent keys essentially at random, recognizing only 50% of the keystrokes correctly. A spelling model integrated into the recognizer can help assuming the typist intended to enter a dictionary word, but then actually hinders entry of other strings. Thus there exists a need in the touchscreen and touch keyboard art for typing recognition methods that are less sensitive to the hand drift and finger placement errors that occur without strong tactile feedback from key centers.
For many years, speech, handwriting, and optical character recognition systems have employed spelling or language models to help guess users"" intended words when speech, handwriting, or other input is ambiguous. For example, in U.S. Pat. No. 5,812,698 Platt et al. teach a handwriting recognizer that analyzes pen strokes to create a list of probable character strings and then invokes a Markov language model and spelling dictionary to pick the most common English word from that list of potential strings. However, such systems have a major weakness. They assume all user input will be a word contained in their spelling or language model, actually impeding entry of words not anticipated by the model. Even if the user intentionally and unambiguously enters a random character string or foreign word not found in the system vocabulary, the system tries to interpret that input as one of its vocabulary words. The typical solution is to provide the user an alternative (often comparatively clumsy) process with which to enter or select strings outside the system vocabulary. For example, U.S. Pat. No. 5,818,437 to Grover et al. teaches use of a dictionary and vocabulary models to disambiguate text entered on a xe2x80x98reducedxe2x80x99 keyboard such as a telephone keypad that assigns multiple characters to each physical key. In cases that the most common dictionary word matching an input key sequence is not the desired word, users must select from a list of alternate strings. Likewise, users of speech recognition system typically fall back on a keyboard to enter words missing from the system""s vocabulary.
Unfortunately, heavy reliance on spelling models and alternative entry processes is simply impractical for a general-purpose typing recognizer. Typing, after all, is the fallback entry process for many handwriting and speech recognition systems, and the only fallback conceivable for typing is a slower, clumsier typing mode. Likewise, personal computer users have to type into a wide variety of applications requiring strange character strings like passwords, filenames, abbreviated commands, and programming variable names. To avoid annoying the user with frequent corrections or dictionary additions, spelling model influence must be weak enough that strings missing from it will always be accepted when typed at moderate speed with reasonable care. Thus a general-purpose typing recognizer should only rely on spelling models as a last resort, when all possible measurements of the actual typing are ambiguous.
Since a typing recognizer cannot depend too much on spelling models, there still exists a need in the touchscreen and touch keyboard art for spelling-independent methods to improve recognition accuracy. The main aspect of the present invention is to search for the geometric pattern of keys that best matches the geometric pattern of a touch sequence, rather than just searching for the key closest to each touch. This method improves recognition accuracy without any assumptions about the character content being typed.
According to this aspect of the invention, touch or finger stroke coordinates reported by a sensing device and key coordinates from a key layout feed into a typing recognizer module. The typing recognizer then hypothesizes plausible sequences of keys by extending existing sequences with keys that are within the immediate neighborhood of the newest finger touch. It can also hypothesize home row key locations for touches caused by hands resting on or near the home row keys. For each hypothesized sequence, the typing recognizer computes separation vectors between the layout position of successive keys in the sequence. The typing recognizer also computes separation vectors between successive touch positions in the touch sequence. Each key sequence is evaluated according to a pattern match metric that includes not only the distance between each finger touch and the corresponding key but also how closely the separation vectors between successive touches match the separation vectors between successive keys. The hypothesized sequence with the best cumulative match metric is transmitted to the host computer, possibly replacing an older, higher cost partial sequence that was transmitted previously.
It is therefore an objective of this invention to provide typing recognition methods that overcome the shortcomings of the related touchscreen and touch keyboard art.
A primary objective of the present invention is to recognize typing accurately even when lack of tactile key position feedback leads to significant hand and finger drift.
Yet another objective of this invention is to improve typing recognition accuracy without excessive dependence on spelling models.
A further objective of this invention is to disambiguate typing as much as possible with measurements of its geometric pattern before falling back on a spelling model to resolve any remaining recognition ambiguities.
A secondary objective of this invention is to beneficially incorporate key/hand alignment measurements from resting hands into recognition decisions without explicitly shifting the key layout into alignment with the resting hands.